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Abstract 

Faculty members at Idaho State University evaluated the use of case analysis as an 
accountability measure for demonstrating teacher candidates’ abilities to meet targeted course, 
institutional, and state teaching standards for student motivation and classroom management. 
Thirty-four teacher candidates completed a case analysis assessment. Predictive validity was 
explored by comparing the teacher candidates' case analysis scores with the teaching observation 
scores they received during a field-based internship. The findings support the generalizability of 
case analysis ratings made on the basis of a standard-based scoring rubric, and the ratings were 
shown to differentiate levels of performance in accordiance with those standards. Support for the 
content validity of the case analysis assessment was obtained by appying criteria forjudging the 
content representativeness of performance assessments suggested by Crocker (1997), including 
realism, frequency, necessity, and importance of the targeted teaching behaviors to actual practice. 
The findings also support one aspect of the predictive validity of the case analysis assessment for 
appraising candidates’ overall ability to manage a public school classroom. 
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Using Complex Case Analysis to Make Visible the Quality of Teacher Candidates 

Teacher education programs across the nation are being expected to meet higher 
requirements for accountability than ever before. In response to the call from The National 
Commission on Teaching and America’s Future (What Matters Most, 1996), which asked states to 
set high standards for what teachers should know and be able to do, many states have recently 
adopted rigorous new standards for beginning teacher certification. These same states are now 
searching for ways to obtain clear and convincing evidence that prospective teachers are able to meet 
the newly set standards. Consistent with this move to higher standards, the National Council for 
Accreditation of Teacher Education (NCATE, 2000) has established new unit accreditation 
standards requiring teacher education programs to document the ability of their graduates to meet 
institutional and state standards. To effectively respond to these mandates and demands for 
accountability, institutions that prepare teachers must develop and implement assessment systems 
that yield defensible evidence regarding their graduates’ ability to meet institutional and state 
standards. Concomitantly, there is considerable concern in the education community about over- 
reliance on the use of paper-and-pencil tests for this purpose. Thus, there is a need to explore 
alternative methods of assessment that yield credible evidence of accountability for teacher 
candidates’ performance with respect to beginning teacher certification standards. 

At Idaho State University, we are exploring teacher candidates’ abilities to analyze complex 
cases as an alternative accountability measure. Cases are descriptions of highly contextualized and 
complex activities representing an authentic snapshot of reality at a particular moment in time, such 
as how a teacher has managed a difficult class over an observed period of time. 

Cases try to capture the richness of lived experience and the dynamic complexity of real life. 
To successfully analyze an assigned case, the reader (a teacher education candidate in this instance) 
must place himself or herself in the position of the main actor in the case and imagine himself or 
herself confronted with the same decisions or difficult problems. Next, the reader must analyze the 
actions of all of the participants in the case as they relate to the main issues of the case. This 
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requires the ability to apply reasoning and principles to the complexities of the case (Forman & 
Rymer, 1999). Finally, the reader is asked to consider how they might do things differently and 
why. 

Responding to a case requires knowledge of appropriate alternatives based on theories, 
principles, and evidence, and the ability to formulate a deliberate action plan that applies this 
knowledge to the context of the case (Forman & Rymer, 1999, Merseth, 1991). Because a case 
analysis requires evaluative choices and confronts beginning teachers with the need to justify ethical 
and moral decisions, a cases analysis may also serve as a means for evaluating teacher education 
candidates’ performance with respect to dispositional standards (Luckowski, 1997). 

Case method teaching has long been associated with professional schools in business, law, 
and medicine (Forman & Rymer, 1999). In the last decade, many educators have advocated the use 
of cases as a way to enhance the quality of teacher education (e.g., Bliss & Mazur, 1998; Colbert, 
Trimble & Desberg, 1996; Merseth, 1991). Case methods have also been successfully utilized in 
teacher education programs to help teacher candidates to better understand the complex nature of 
classroom management (Adler, 1996). If cases can be used as instructional tools, it is a logical 
extension to consider using them for the purpose of assessment as well. 

We could find little research, however, to support the use of case analysis as an assessment 
measure (other than the use of a case write-up to demonstrate understanding of the case itself). We 
were also disappointed to find there was a paucity of evidence to support the claim that analyzing 
cases improves teaching performance (see however, Kleinfeld, 1998). Thus, a major purpose of our 
investigation was to explore whether or not performance on a case analysis could provide credible 
evidence of candidate performance relative to standards and whether or not that evidence generalized 
to actual teaching situations. 

Our study addresses the development of guidelines and a standards-based scoring rubric 
for a case analysis portfolio entry produced by teacher education candidates as partial 
documentation of their ability to meet program and state standards related to classroom 
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management and student motivation. We also examined the inter-rater reliability, content 
representatives (content validity), and one aspect of the predictive validity of our case analysis 
assessment. Among the questions addressed were: Can teacher education faculty agree sufficiently 
about the important issues in a case to be able to determine whether or not teacher education 
candidates have responded effectively? Can case analyses be used to differentiate the performance 
of teacher education candidates with respect to program and state teaching standards? Can these 
differences be reliably scored by a panel of raters? If the variability in performance can be reliably 
scored, does it predict actual practice when candidates are confronted with the problems of 
classroom management during an early internship experience? 

Method 

Teacher Candidate Participants 

Eighteen elementary and 1 6 secondary education teacher candidates who agreed to 
participate in this study completed the case analyses. There were 8 males and 26 females. One 
candidate was Hispanic; the remaining candidates were non-Hispanic Caucasians. Their average age 
was M =27 years old (varying from 21 to 44 years old). All of the teacher candidates were 
enrolled in a junior-level course on student motivation and classroom management during the 2001 
spring semester. All teacher candidates are required to take this course and complete the case 
analysis prior to admission to student teaching. The candidates received the cases and guidelines 
for completing the case analysis from their course instructors. They completed the cases 
independently and without the direct assistance of their instructors. 

Case Analysis Assessment 

Faculty members who taught the motivation and management course identified five target 
standards the teacher candidates were to demonstrate through the completion of the case analysis. 
The case analysis standards (see Appendix A) were designed to reflect behaviors teachers might use 
in a problem-solving approach to motivation and management issues and were aligned with Idaho 
State University’s Beginning Teacher Core Standards and Indicators (College of Education, 
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1995) and the Idaho Core Teacher Standards (Idaho Board of Education, 2000). 

The same faculty members also developed the Case Analysis Guidelines (See Appendix B). 
The guidelines contained a description of the case analysis assignment and directions for 
completing the case analysis matrices and formatting the written product. The cases were selected 
from the text, Case Studies for Teacher Problem Solving (Silverman, Welty, & Lyon, 1996). 
Separate cases focusing on broad issues of motivation and management, not just issues of order 
and control, were selected for elementary and secondary education teacher candidates. The case 
selected for elementary education candidates was Case 1 : Marsha Warren. The case selected for 
secondary education candidates was Case 12:Frank Oakley: The Classroom. 

The case analysis assessment required the teacher candidates to read and to analyze a 
complex caset. After they read their assigned case, the candidates identified the significant issues 
related to student motivation and the creation of effective classroom environments, and issues 
related to student conduct and misbehavior. Next, for each issue they: (1) identified the actions 
taken (or inaction) by the teacher or students, (2) analyzed the issues and actions according to the 
concepts and principles presented in their motivation and management course, (3) provided 
alternative strategies, and (4) wrote a rationale for each alternative strategy citing concepts and 
principles taught in their motivation and management course. The teacher candidates recorded their 
analyses on the matrices presented in Appendix B. 

Case Analysis Scoring Rubric and Scoring Procedures 

Using the case analysis standards a scoring rubric was developed to evaluate the teacher 
candidates' performance and to provide specific feedback to the candidates regarding their 
performance on each of the targeted standards (See Appendix C). The scoring rubric specified 
each standard along with a set of indicators for each standard. The indicators served as the criteria 
forjudging performance relative to the standards (See Appendix C). Each of the five standards was 
rated on a 4-point scale: 0 = Standard Not Met ; 1 = Standard Partially Met', 2 = Standard Met\ 3 = 
Exemplary Performance. A total score was computed by summing the scores across the five 
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standards measured on the rubric. 

Case Analysis Raters 

Four faculty members with experience teaching the motivation and management course 
scored the N = 34 case analyses. Prior to scoring the cases, the faculty members assembled to 
review the cases and reach a consensus on the key issues presented in each case. The issues 
identified were associated with major topics taught in the candidates’ course on motivation and 
management. The faculty members also spent time discussing application of the standards-based 
scoring rubric and the indicators for each standard. Each faculty member rated independently all 34 
case analyses collected for this study. All identifying information was removed from each case 
analysis prior to distributing them to the raters. 

Validity Assessment Panel 

Five teachers with experience teaching elementary school (grades K - 6) and five teachers 
with experience teaching in secondary schools (grades 7-12), served as the validity assessment 
panel for this study. Some of the teachers had experience teaching in both elementary and 
secondary schools. The teachers had an average of M= 15.7 years (ranging from 3-25 years) of 
teaching experience. Five of the teachers had bachelor degrees and 5 of the teachers had masters 
degrees. Three of the teachers were National Board Certified. 

Content Validity Questionnaire 

A questionnaire based on a performance assessment validation process developed by 
Crocker (1997) was used to assess the content representativeness of the case analysis assessment. 
The questionnaire first asked the validity assessment panel to evaluate the degree of alignment 
between the Case Analysis standards, guidelines, and scoring rubrics. The panel members were 
next asked to evaluate the degree to which Crocker’s criteria for content representativeness were 
met. The first criterion was how frequently beginning teachers would be expected to engage in the 
teaching behaviors targeted in the case analysis in actual job performance. This was evaluated on a 
five-point scale of (1) never, (2) monthly, (3) weekly, (4) daily, or (5) hourly. The second criterion 
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was the importance or criticality of those behaviors to actual teaching. This was judged on a four- 
point scale of (1) not at all important, (2) somewhat important, (3) important, or (4) absolutely 
important. The third criterion was the realism of the tasks to actual classroom practice. This 
criterion was also judged on a four-point scale from (1 )not at all realistic to (4) absolutely 
realistic. The fourth criterion was the degree to which the tasks represented the targeted standards. 
The panel members were asked to judge this on a four point scale of: (1) not at all representative, 
(2) somewhat representative, (3) representative, or (4) very representative. The validity 
assessment panel was also asked to assess the degree to which the tasks performed while 
completing the case analysis and the performance indicators on the scoring rubric assessed any of 
the ten Idaho Core Teacher Standards (Idaho Board of Education, 2000). Finally, the assessment 
panel was asked to respond to four overall questions about the validity of the case analysis 
assessment as a performance assessment and accountability measure . 

Procedures for Assessing Content Validity 

The assessment panel completed the content validity questionnaire during a two-hour 
session after school. The panel members first read the case analysis standards, guidelines, and 
scoring rubric. Next, they read either the elementary teacher case or the secondary teacher case 
selected for use in this study. After reading the cases, the panel members then read several case 
analyses produced by the teacher candidates. Finally, they responded to the items on the content 
validity questionnaire and completed a short demographic questionnaire about their own 
background and teaching experience. 

Procedures for Collecting Predictive Validity Data 

Our teacher education candidates take our Motivation and Management course prior to 
completing a semester long, half-time internship in the public schools. This initial internship is 
attached to a junior-level course, titled Planning, Delivery and Assessment. The instructors for this 
course supervise the teacher candidates during their initial field placement. The teacher candidates 
are expected to carry forth the concepts and theories learned in our course on Motivation and 
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Management and to apply them during this initial internship experience. To assess the performance 
of the candidates during the field placement, the supervisors make multiple observations of each 
candidate’s teaching performance using a standard form called the Teaching Performance Scoring 
Rubric. This rubric was previously developed by our teacher education faculty (College of 
Education, 2000). 

The Teaching Performance Scoring Rubric has a sub-scale with standard and indicators for 
Management of the Learning Environment (MLE). The MLE standard (College of Education, 
2000, p. 2) is “The teacher creates and maintains a safe and effective learning environment.” The 
indicators (College of Education, 2000, p. 2) of the standard are: (1) “Creates a smoothly 
functioning learning environment in which students work collaboratively and independently to 
complete purposeful learning activities and in which students assume responsibility for themselves 
and one another;” (2) “sets and maintains clear expectations for communication and behavior;” 

(3) “selects and uses effective strategies for handling behavior problems;” (4) “organizes and 
manages resources of time, space, activities, and attention to create a safe and effective learning 
environment.” This standard was rated on a three-point scale of: (0) Standard Not Met, (1) 
Standard Partially Met, or (2) Standard Met for each teaching observation. Our program supposes 
our teacher candidates will have received much of the knowledge, skills, and dispositions needed to 
achieve this standard from our Motivation and Management course. 

Of the 34 teacher candidates participating in this study, 20 were enrolled in the Planning, 
Delivery and Assessment course during the fall of 2001. As a result, the course instructors were 
asked to send us the Management of the Learning Environment sub-scale scores for these 
candidates after their initial observation of their teaching performance. These observations were to 
be made when the candidates had initial charge of the activities of the entire class. We received 
MLE scores for 19 of the 34 teacher candidates who completed case analyses for this study. 

Design & Data Analysis 

To assess the consistency of the scores on our analytic scoring rubric made by our panel of 
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raters, we applied concepts from Generalizability Theory (Shavelson & Webb, 1991). We 
computed a dependability coefficient for absolute decisions for the case analysis scores using 
formulas provided by Shavelson and Webb (1991). Generalizability Theory provides a summary 
coefficient reflecting the level of dependability of raters that is similar to a reliability coefficient 
(Shavelson & Webb, 1991). We also kept track of and analyzed the amount of time it took to score 
the two sets of case analyses. Predictive Validity was determined by computing the relationship 
between the analytic scores on the case analyses and the Management of the Learning Environment 
sub-scale scores these same teacher candidates received in the Fall Semester 2001 during their first 
field placement (EDUC 309 Planning, Delivery & Assessment 6 credits.) The level of significance 
for all statistical analyses was set at .05. 

Results 

Interrater Reliability 

We computed total score dependability coefficients for absolute decisions based on our 
case analysis scoring rubric for four independent raters using formulas provided by Shavelson and 
Webb (1991). The repeated measures analysis of variance for the overall effect of rater across all 
34 case analyses was not statistically significant, F( 3, 99) = 1 .34, MSE = 2.82, p = .27. Examining 
the effect of rater separately for the elementary school cases analyses in - 18) and the secondary 
school case analyses (n = 16) also revealed no significant effect for rater, F( 3, 51) = .09, MSE = 

3.1 1, p = .97 and F(3,45) = 2.67, MSE = 2.48, p = .06, respectively. 

Table 1 presents the variance components used in the formulas for computing the 
dependability coefficients. The four rater coefficient of dependability for all 34 case analyses was 
.92. For the 18 elementary cases analyses, the four rater coefficient of dependability was .87 and 
for the 16 secondary case analyses, the four rater coefficient of dependability was .91 . The single 
rater coefficients of dependability were .75, .77, and .72 respectively. Adjusting the number of 
raters included in the formulas revealed dependability coefficients from .84 to .87 could be achieved 
with as few as two raters. Overall, these findings suggest case analyses can be scored with 




11 



Using Complex Case Analysis 1 1 



sufficient inter-rater agreement to make decisions regarding the quality of teaching performance. 
The above findings also support the use of the average rating of the four raters as benchmark scores 
for future case analyses written for these cases and for the training and calibrating of future raters. 



Insert Table 1 about here 



Differentiation of Teaching Competence 

Based on the average rating each case analysis received from our panel of four raters, we 
classified our candidates’ performances into different levels of competence according to the degree 
to which each performance met our standards. To be judged and classified as Proficient on all 
standards, our candidates must have received an average total rating of 10 or higher. The course 
instructors agreed that an average total performance of 13 or higher would be regarded as 
Exemplary performance. Thus, the Proficient category ranged from an average total performance of 
10 to 12.9 and the Exemplary category ranged from an average score of 13 through a perfect 
average score of 15. Case Analysis performances receiving average total scores between 5 and 9.9 
points were classified as Developing, because in general the standards were only partially met. 

Cases Analysis performances receiving average total scores below 5 were classified as Beginning, 
because in general the standards were not met. Using the above criteria, n = 4(11 .8%) of the case 
analysis performances were judged to be at the Beginning level, n = 19 (55.9%) of the 
performances were judged to be Developing, n = 8 (23.5%) were judged to be Proficient, and n = 3 
(8.8%) were judged to be Exemplary. Hence, our results show the analysis of complex cases can 
be use to differentiate teacher candidates’ performances into four distinct levels along a 
developmental continuum from beginning level to highly proficient performance on the basis of the 
degree to which our teacher candidates have demonstrated their ability to meet targeted program and 
state teaching standards. 

Time Required to Score Case Analyses 
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We next considered the amount of time necessary to reliably score case analysis 
performances using our Case Analysis Rubric. The average time for scoring the 34 cases analyses 
by the four raters was M = 7.9 minutes with a standard deviation of SD = 3.5 minutes. One of the 
raters took consistently longer to score than the others with M = 12.5 minutes as compared to M = 
7.0 minutes, M = 6.2 minutes, and M = 6.0 minutes for the other three raters respectively. 
Fortunately, the time spent scoring was not statistically significantly correlated with the case 
analysis scores, r = .15, n = 136 (34 cases by 4 raters), p = .08. These data show the time it takes 
to reliably score cases analysis performances is well within a range that is realistic and practical for 
accountability assessment purposes. It should be noted, however, these times are for scoring time 
only and did not include the time it would take to provide written comments or corrective advise to 
each candidate about his or her performance. 

Content Validity 

To support the validity of our Case Analysis assessment for making judgments about 
teacher candidates’ performance with respect to program standards and state certification 
requirements, we applied criteria suggested by Crocker (1997) forjudging the content 
representativeness of performance assessments. Our first consideration was the adequacy of the 
alignment among the Case Analysis Guidelines, scoring rubric, and targeted standards (e.g, the 
assessment domain). Table 2 presents the judgments made by our panel of expert raters. All the 
raters thought there was a high degree of alignment between the Case Analysis Guidelines and the 
targeted standards. Ninety percent of the raters thought there was high alignment between the Case 
Analysis Guidelines and the scoring rubric. All of the raters also indicated that there was a high 
degree of alignment between the targeted standards and the scoring rubric. 



Insert Table 2 about here 
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We next asked our panel of experts to consider the degree to which the case analysis 
performance tasks reflected and represented each of the targeted standards. Table 3 presents the 
number and percentage of the raters judging each targeted standard to be represented by the tasks 
required in the case analysis. It is clear from looking at table 3, the tasks were considered to be 
representative of the targeted standards. For all but one of the standards, 100% of the raters 
thought the tasks were representative or very representative. For the standard of using the 
professional literature to provide a rationale and research support for practices, 90% of the 
assessment panel thought the tasks required by the case analysis were representative or very 
representative of that standard. 



Insert Table 3 about here 



We also asked the assessment panel to rate the importance of the teaching behaviors 
targeted by the case analysis to the success of our candidates as teachers. Table 4 presents the 
number and percentage of the panel members judging each targeted behavior to be of importance 
All of the targeted behaviors were rated as important or very important with 70% or more of the 
assessment panel rating them as very important. These results support the criticality criteria for 
valid performance assessment. 



Insert Table 4 about here 



The frequency of the teaching behaviors in job performance was assessed by asking our 
assessment panel how often they would expect a beginning teacher to engage in each of the tasks 
required by the case analysis during the course of his or her professional practice. Table 5 presents 
the number and percentage of the panel members indicating how frequently from never to hourly 
they would expect each of the targeted teaching behaviors. As can be seen from the table, all but 
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one of the targeted behaviors was considered to be a high frequency activity for teachers with 70% 
or more of the raters endorsing a frequency level of daily or hourly. The targeted behavior 
considered to be of a lower frequency (i.e., monthly) by half of the raters was the behavior of 
providing a rationale for management practices to administrators, parents, other teachers, or 
students. Comments related to this item indicated the assessment panel had difficulty separating the 
different audiences for the rationale mentioned in this item. In general, the raters thought providing 
a rationale for actions to students was a high frequency activity, but providing a rationale to 
administrators, parents, and other teachers would happen less often. All together, these results 
support the frequency criteria necessary for valid performance assessment. 



Insert Table 5 about here 



The realism of the tasks required by the case analysis was also assessed by our assessment 
panel. Table 6 presents the number and percentage of the raters judging the tasks to be realistic. 
Across all of the tasks, 80% or more of the panel members thought the tasks were realistic or very 
realistic to success as a classroom teacher. Amazingly, 80% of the raters thought adhering to 
format requirements and expressing ideas using standard English free from mechanical and 
spelling errors was a very realistic requirement for success as a classroom teacher. As one teacher 
on the panel stated, “Teachers are required to follow the format and to complete forms properly all 
of the time and we are expected to do so using standard English.” Together, these results provide 
favorable support for meeting the realism criteria for authentic performance assessment. 
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Insert Table 6 about here 



Additionally, we asked the assessment panel to respond to four questions concerning the 
validity of the case analysis assessment overall. The first question was, “Overall, does the Case 
Analysis measure knowledge and skills that are necessary for a beginning teacher?” The results 
were 10% (n = 1) of the raters said they were “somewhat necessary,” 30% ( n = 3) said they were 
“necessary,” and 60% (n= 6) said the were “absolutely necessary.” The second question asked, 
“Overall, how critical to the practice of a beginning teacher are the skills and strategies the Case 
Analysis requires teacher candidates to demonstrate?” The response percentages were the same. 
Ten percent of the raters ( n =1) said “somewhat critical,” 30% ( n = 3) said “critical,” and 60% ( n 
= 6) said “absolutely critical.” The third question asked, “Overall, does the case analysis present 
teacher candidates with realistic situations similar to ones they might encounter in professional 
practice as a teacher?” Here 100% of the panel members said the situations were realistic (n = 1) 
or very realistic ( n = 9). The final question was, “Overall, how appropriate is it to use the case 
analysis assessment as one measure of a beginning teacher’s competency?” Two of the panel 
members (20%) thought the case analysis was “somewhat appropriate,” four of the panel members 
(40%) thought it was “appropriate,” and four of the panel members (40%) thought it was 
“absolutely appropriate.” The panel members who thought it was only somewhat appropriate said 
the performance levels might be too much to expect from a beginning teacher. Together, the 
responses to these questions support the overall validity of the use of a complex case analysis as a 
performance assessment forjudging beginning teaching competence with respect to targeted 
program standards. 

Our final consideration was the extent to which the performances required by the case 
analysis directly assessed any of the Idaho Core Teacher Standards (Idaho State Board of 
Education, 2000). The teacher education program standards targeted by the case analysis 
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assessment were directly linked to two of the ten Idaho standards and probably were implicitly 
related to several others. However, the case analysis was not designed to be a direct measure of all 
of the Idaho standards. Accordingly, we asked our panel of experts to indicate the extent to which 
the tasks required for the case analysis measured the Idaho Core Teacher Standards (Idaho State 
Board of Education, 2000) using a scale of (1) Not At All, (2) Implicitly, and (3) Directly. Table 7 
presents the number and percentage of the responses for each standards. Clearly, some Idaho 
standards were considered to be directly measured, while others were judged to be implicitly 
measured or not at all. Importantly, both of the standards directly targeted were thought to be 
directly measured by 100% and 90% of the our assessment panel members. These findings 
support the validity of our case analysis assessment as a direct measure of our candidates’ ability to 
meet some of the Idaho Core Teaching Standards. Our findings also suggest complex case 
analyses supply credible evidence of candidate performance for accountability purposes. 

Predictive Validity 

We assessed the predictive validity of the cases analysis scores by correlating both the 
average total scores on the case analysis and the average scores across raters for each case analysis 
standard with the teacher candidates’ Management of the Learning Environment (MLE) sub-scale 
scores. The MLE scores were not significantly correlated with the average total scores on the case 
analysis, r = .45, n = \9,p = .055. The relationship was positive, but did not reach statistical 
significance given the small number of candidates for whom we had obtained both scores. 

However, the MLE scores (n = 19) were found to be significantly correlated with the scores on the 
Rationale standard, r = .46, p = .048. This means the candidates’ use of the professional literature 
on motivation and management to provide a theoretical rationale and research support for 
educational practices when completing a case analysis positively predicted MLE scores when our 
candidates were later observed managing students in a public school classroom. No other 
significant correlations were found between the scores on the case analysis standards (n = 19) and 
the MLE scores, r = .28, p = .26 for Issue Identification, r = .34, p = .1 5 for Analysis, r = .34, p = 
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. 1 6 for Alternative Strategies, and r = .36, p = . 1 3 for Format and Usage. However, the direction 
of all of these correlations was in a positive direction, and merit further investigation when we are 
able to collect scores for a larger number of our candidates. 

Discussion 

We examined the use of case analysis as an assessment tool because of the potential for 
cases to provide a rich and realistic context for the appraisal of teacher candidates’ application of 
the shared knowledge, skills, and dispositions valued by a professional teaching community. Our 
cases were selected because they could be thematically linked to the motivation and management 
standards of our program. Our first concern was whether teacher education faculty members could 
agree sufficiently about the important issues in a case to be able to determine whether or not the 
teacher education candidates had responded effectively to them. We also wanted to know whether 
our teacher candidates could analyze an assigned case from the perspective of the professional 
knowledge base, and not just their own personal knowledge and experiences, and produces 
responses that could be reliably scored. 

The findings of our study demonstrate teacher education faculty members can reach 
consensus about the significant issues of a complex case as they relate to course topics and they 
can score sets of case analyses with a high degree of inter-rater agreement. We found the overall 
four rater dependability of the case analysis scores to be .92. We also found dependability 
coefficients with magnitudes of .84 to .87 could be achieved with as few as two raters. This means 
a very high proportion of the differences among the scores on our case analysis rubric reflect 
differences in actual levels of performance (absolute decisions) that can be generalized over raters. 
From a practical standpoint, this also means absolute decisions can be made about the performance 
levels of teacher candidates using only a few raters. 

One reason we obtained such high inter-rater agreement may have been the highly 
structured nature of our case analysis assessment. The teacher candidates were limited to a 
maximum of ten pages of content. Eight of those pages were constrained to very structured 
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matrices. The matrix format prompted the teacher candidates to supply specific types of 
information within the space afforded by the format. The candidates responses were then scored 
using a common rubric with clearly articulated performance indicators linked to standards. This 
approach is quite different from how others have assessed candidates understanding of a case. 
Assessment approaches have included activities such as responses to essay questions (Vesper & 
Adams, 1972), presentations in a public forum (Kuntz & Hessler, 1998; McNergney, Herbert & 
Ford, 1994), the develop of concept maps (Adler, 1996), case write-ups from personal and 
professional knowledge (Mostert & Sudzina, 1996), and case analysis papers responding to a 
structured set of questions (Kleinfeld, 1991). Each of these approaches may provide different 
insights into the knowledge candidates have gain from their examination of a case. In this study, we 
have shown teacher candidates’ responses to a structured case analysis assessment can be scored 
with sufficient consistency to make absolute decisions about their performance levels with respect 
to program and state teaching standards. 

Validity of Case Analysis Assessment 

The next major concern of this study was whether completion of a case analysis would be 
judged to provide valid and credible evidence of teacher candidate performance relative to program 
and state standards. This study found support for the validity of a case analysis assessment of 
teaching performance focused on standards with respect to student motivation and classroom 
management. According to our validity assessment panel, the scores derived from our case analysis 
rubric meet the criteria for content representativeness stated by Crocker (1997), including realism, 
frequency, necessity, and importance (or criticality). The case analysis assessment was also found 
to appraise the application of knowledge, skills, and dispositions that were representative of those 
used in actual teaching situations. Although slightly removed form actual classroom practices, the 
analysis of a complex case was judged to provide teacher candidates with a valid opportunity to 
demonstrate their abilities to meet targeted program and state teaching standards. 

Differentiation of Levels of Competence 
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To provide credible evidence for making judgments about teacher candidates’ performance 
with respect to program standards and state certification requirements, case analyses must also be 
shown to differentiate levels of competence in accordance with those standards and requirements. 
Our findings demonstrated candidates’ performances can be differentiated along a continuum from 
beginning to exemplary on the basis of whether or not the standards targeted by the case analysis 
were met. Importantly, slightly less than a third of our candidates were judged to be proficient or 
exemplary with respect to the standards set for our cases analysis assessment. This finding 
suggests we need to determine ways to improve our program if we are serious about helping our 
candidates achieve the program standards with respect to motivation and management. However, 
this also means our case analysis assessment does provide a clear way to measure important aspects 
of what our teacher candidates know and can do. Thus, our case analysis assessment will also serve 
as an effect tool to measure whether or not our program has improved. It should be noted, however, 
the teacher candidates for this study were volunteers; we do not yet know whether these percentages 
are representative of all the teacher education candidates in our program. Nevertheless, we have 
established this important first step to the use of case analysis assessment to document candidate 
performance levels. 

Predictive Validity 

Kleinfeld, (1998) has shown the predictive validity of the use of cases-based instruction on 
the cross-cultural teaching skills of teacher education candidates. Our study indicates the ability to 
analyze a case effectively by providing a rationale for suggested alternative decisions and strategies 
predicts the initial ability of teacher candidates to manage a public school classroom during 
instruction. Together, these studies support the idea that cases provide an authentic means of 
assessment that predicts candidates’ abilities to teach. The evidence at this point is modest. We 
only found a statistically significant relationship for the one standard addressing the ability to 
provide a rationale for suggested motivation and management practices (although all of the 
correlation coefficients from this small study were in a positive direction). We are currently in the 
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process of refining this part of our investigation and conducting a larger study. Still, the fact that 
we found a predictive link at all is important, because according to Darling-Hammond and Snyder 
(2000, p. 524), “studies of the predictive validity of traditional paper-and-pencil tests of teaching 
(for example, the National Teacher Examinations) have found little evidence that such tests are 
correlated with teacher ratings or teachers’ classroom effectiveness.” Alternatively, our study does 
provide some initial modest support for the predictive validity of an authentic and context rich form 
of teacher assessment via case analysis. 

Of considerable interest is our finding that it was candidate differences in their ability to 
generate a rationale for proposed alternative actions linked to course concepts and principles that 
best predicted later ratings of their actual classroom management. One of the claims most often 
made on behalf of case-based instruction is that it “encourages reflective practice and deliberate 
action ” (Merseth, 1991, p. 16). When responding to cases candidates must build and evaluate their 
own action plans in light of contemporary theories and principles of best practice in a way that 
integrates their theoretical, practical, and contextual knowledge. The ability of candidates to perform 
such integrations may be an important factor to their success as teachers. As one of our assessment 
panel members commented, “one thing to stress in preparation of education students is to be 
reflective and to understand that there is a rationale behind teaching decisions. The stronger and, in 
my opinion, more marketable teacher education graduates are those who look beyond the 
instructional materials and the acquired management practices to analyze and to reflect upon the 
rationale behind their choices as an educator.” Our data support this opinion of a highly 
experienced and National Board Certified classroom teacher. They also support the view that case 
analysis is a viable means to determine which teacher candidates are best able to do this. 

Conclusion 

Our results suggest case analysis has great promise as one alternative to traditional paper- 
and-pencil tests forjudging teacher candidate performance that is benchmarked against standards. 
Because the Idaho Core Teacher Standards (Idaho State Board of Education , 2000) are based on 
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the Interstate New Teacher Support and Assessment Consortium standards (INTASC, 1992), our 
findings are also likely to generalize to other teacher education programs whose state and program 
standards are based on or similar to the INTASC standards. Accordingly, data from case analyses 
might be included as one method for providing evidence of candidates’ abilities to meet such 
standards required for unit accreditation (National Council for Accreditation of Teacher Education, 
2000 ). 




22 



Using Complex Case Analysis 22 



References 

Adler, S. A. (1996). On case method and classroom management. Action in Teacher 
Education, 18(3), 33-43. 

Bliss, T., & Mazur, J. (1998). Secondary and middle school teacher in the midst of reform: 
Common Thread Cases. Upper Saddle River, NJ: Prentice-Hall. 

Colbert, J. A., Desberg, P., & Trimble, K. (eds.) (1996). The case for education: Contemporary 
approaches for using case methods. Boston: Allyn and Bacon. 

College of Education (1995). Beginning teacher core standards and indicators. Unpublished 
Manuscript, Idaho State University, Pocatello, Idaho. 

College of Education (2000). Teaching Performance Scoring Rubric. Unpublished Manuscript, 
Idaho State University, Pocatello, Idaho. 

Crocker, L. (1997). Assessing content representativeness of performance assessment 
exercises. Applied Measurement in Education, 10, 83-95. 

Darling-Hammond, L., & Snyder, J. (2000). Authentic assessment of teaching in context. Teaching 
and Teacher Education, 16, 523-545. 

Forman, J., & Rymer, J. (1999). Defining the genre of the “case write-up.” The Journal of 
Business Communication, 36(2), 103-133. 

Idaho State Board of Education (2000). Idaho standards for initial certification of professional 
school personnel. Boise, ID: Idaho State Department of Education. 

Interstate New Teacher Assessment and Support Consortium. (1992). Model standards for 
beginning teachers licensing and development: A resource for state dialogue. Washington, 

DC: Council of Chief State School Officers. 

Kleinfeld, J. (1991, April). Changes in problem solving abilities of student taught through case 
methods. Paper presented at the annual meeting of the American Educational Research 
Association, Chicago, Illinois. (ERIC Document Reproduction No. ED 334 154). 

Kleinfeld, J. (1998). The use of case studies in preparing teachers for cultural diversity. Theory into 




23 



Using Complex Case Analysis 23 



practice, 37(2), 140-147. 

Kuntz, S., & Hessler, A. (1998). Bridging the gap between theory and practice: Fostering active 
learning through the case method. Paper presented at the annual meeting of the Association 
of American Colleges and Universities, Washington, DC. (ERIC Document Reproduction 
No. ED 420 626). 

Luckowski, J. A. (1997). A virtue-centered approach to ethics education. Journal of Teacher 
Education, 48(4), 264-270. 

McNergnery, R. F., Herbert, J. M., & Ford, R. E. (1994). Cooperation and competition in case- 
based teacher education. Journal of Teacher Education, 45(5), 339-345. 

Merseth, K. K. (1991). The case for cases in teacher education. Washington, DC: American 
Association for Higher Education and the American Association of Colleges for Teacher 
Education. 

Mostert, M. P., & Sudzina, M. R. (1996, February). Undergraduate case method teaching: 
Pedagogical assumptions vs. the real world. Paper presented at the annual meeting of the 
Association of Teacher Educators, St. Louis, Missouri. (ERIC Document Reproduction No. 
ED 395 900). 

National Commission on Teaching and America's Future. (1996). What matters most: 

Teaching and America's Future. New York: Teachers College. 

National Council for Accreditation of Teacher Education. (2000). NCATE 2000 Unit 
Standards. Washington, DC: Author. 

Shavelson, R. J., & Webb, N. M. (1991). Generalizability theory: A primer. Newbury Park, 

CA: Sage. 

Silverman, R., Welty, W. M., & Lyon, S. (1996). Case studies for teacher problem solving (2nd 
ed.). New York: McGraw-Hill. 

Vesper, K. H., & Adams, J. L. (1972). Measuring change produced by case method instruction. 
Engineering Education, 27-40. 




24 



Using Complex Case Analysis 24 



Table 1 

Estimates of Variance Components of the Person and Rater Facets for the Case Analysis Ratings 



Source 


Overall 


Variance Components 
Elementary Case 


Secondary Case 


Person 


8.561 


9.989 


7.217 


Rater 


.027 


-.157 


.259 


Residual 


2.822 


3.106 


2.481 
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Table 2 

Number and Percentage of Expert Raters Indicating Alignment Between Case Analysis Guidelines, 
Targeted Standards and Scoring Rubric (N = JO) 



Alignment Considerations 


Degree of Alignment 


Poor 

1 


Low 

2 


Moderate 

3 


High 

4 


Alignment of the Case Analysis Guidelines 
with the targeted standards 








10 

100% 


Alignment of the Case Analysis Guidelines 
with the scoring rubric 






1 

10% 


9 

90% 


Alignment of the scoring rubric with the 
targeted standards 








10 

100% 
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Table 3 

Number and Percentage of Expert Raters Indicating the Degree to Which the Tasks Required by 
the Case Analysis Reflect and Represent the Targeted Standards (N = 10) 





Degree of Representativeness 


Targeted Standards 


Not at all 


Somewhat 


Representative 


Very 


Representative 


Representative 


Representative 




i 


2 


3 


4 


The teacher considers different 






4 


6 


theories of motivation and 






40% 


60% 


management when confronted with 
classroom motivation and 










management problems in order to 
identify all significant issues and 
aspects relevant to both simple and 
complex problem situations. 










The teacher analyzes teacher-student 






2 


8 


interactions in complex motivation 
and management situations using 
theories of motivation and 






20% 


80% 


management to identify needed 
adjustments to enhance social 
relationships, student motivation and 
engagement in learning, or to maintain 
a safe and effective learning 
environment. 










The teacher selects and uses effective 






3 


7 


strategies drawn from the professional 
literature for handling motivation and 
management problems and to make 
adjustments to practice. 






30% 


70% 


The teacher uses the professional 




i 


1 


8 


literature on motivation and 
management to provide a theoretical 
rationale and research support for 
educational practices. 




10% 


10% 


80% 


The teacher adheres to format 






2 


8 


requirements and models professional 
(standard) English usage in writing. 






20% 


80% 
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Table 4 

Number and Percentage of Expert Raters Indicating the Importance of the Teaching Behaviors 
Targeted by the Case Analysis to Success as a Classroom Teacher(N = 10) 



Teaching Behaviors Targeted 
By Case Analysis Portfolio Entry 


Degree of Importance 


Not at all 
Important 
1 


Somewhat 

Important 

2 


Important 

3 


Very 

Important 

4 


Identify significant issues relevant to simple 
and complex motivation and management 
problem situations. 






3 

30% 


7 

70% 


Analyze teacher-student interactions. 






1 

10% 


9 

90% 


Identify needed adjustments to enhance 
social relationships, student motivation and 
engagement in learning, or to maintain a 
safe and effective learning environment. 






1 

10% 


9 

90% 


Select and use effective strategies for 
handling motivation and management 
problems and to make adjustments to 
practice. 






2 

20% 


8 

80% 


Provide a rationale to administrators, 
parents, other teachers, or students to 
support motivation and management 
practices. 




1 

10% 


2 

20% 


7 

70% 


Adhere to format requirements and model 
professional (standard) English grammar 
and spelling in all written communication 
to administrators, parents, other teachers, or 
students. 






2 

20% 


8 

80% 
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Table 5 

Number and Percentage of Expert Raters Indicating How Frequently They Would Expect a 
Teacher to Engage in the Teaching Behaviors Targeted by the Case Analysis (N = 10) 



Teaching Behaviors Targeted 
By Case Analysis Portfolio Entry 


Never 


Monthly 


Weekly 


Daily 


Hourly 


Identify significant issues relevant to simple and 
complex motivation and management problem 
situations. 






2 

20% 


3 

30% 


5 

50% 


Analyze teacher-student interactions. 






3 

30% 


3 

30% 


4 

40% 


Identify needed adjustments to enhance social 
relationships, student motivation and engagement 
in learning, or to maintain a safe and effective 
learning environment. 






3 

30% 


3 

30% 


4 

40% 


Select and use effective strategies for handling 
motivation and management problems and to 
make adjustments to practice. 






3 

30% 


3 

30% 


4 

40% 


Provide a rationale to administrators, parents, 
other teachers, or students to support motivation 
and management practices. 




5 

50% 




3 

30% 


2 

20% 


Adhere to format requirements and model 
professional (standard) English grammar and 
spelling in all written communication to 
administrators, parents, other teachers, or 
students. 








3 

30% 


7 

70% 
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Table 6 

Number and Percentage of Expert Raters Indicating How Realistic the Tasks Required by the Case 
Analysis Are to Success as a Classroom Teacher (N = 10) 



Tasks Required By the 
Case Analysis Portfolio Entry 


Degree of Realism 


Not at all 
Realistic 
1 


Somewhat 

Realistic 

2 


Realistic 

3 


Very 

Realistic 

4 


Identification of all significant issues and 
aspects relevant to complex motivation and 
classroom management problems presented 
in a realistic case situation. 




2 

20% 


4 

40% 


4 

40% 


Analysis of motivation and management 
problems present in a realistic case situation 
to identify all appropriate and needed 
adjustments warranted by the facts of the 
case. 




2 

20% 


2 

20% 


6 

60% 


Identification of multiple effective 
motivation and management strategies and 
adjustments to practice that address all key 
issues of a realistic case situation. 




1 

10% 


4 

40% 


5 

50% 


Demonstration of knowledge of how to 
implement alternative motivation and 
management strategies and adjustments to 
practice as they apply to a realistic case 
situation. 






2 

20% 


8 

80% 


Provision of broad and convincing 
justifications for stated strategies and 
practices as they apply to a realistic case 
situation, which are supported by an 
appropriate knowledge of the professional 
literature and the terminology used by 
teachers and related educational 
professionals. 






5 

50% 


5 

50% 


Written expression of ideas about a realistic 
case situation models professional 
(standard) English usage, is free of 
mechanical and spelling errors, and adheres 
to format requirements. 






2 

20% 


8 

80% 



O 
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Table 7 

Number and Percentage of Expert Raters Indicating the Extent to Which the Tasks Required by the 
Case Analysis Assess the Idaho Core Teacher Standards (N = 10) 



Standard 


Not at all 


Implicitly 


Directly 


The teacher understands the central concepts, tools of 


2 


7 


1 


inquiry, and structures of the discipline(s) taught and 
creates learning experiences that make these aspects of 
subject matter meaningful to students. 


20% 


70% 


10% 


The teacher understands how students learn and develop, 




7 


3 


and provides opportunities that support their intellectual, 
social, and personal development. 




70% 


30% 


The teacher understands how students differ in their 




5 


5 


approaches to learning and creates instructional 
opportunities that are adapted to learners with diverse 
needs. 




50% 


50% 


The teacher understands and uses a variety of 


2 


3 


5 


instructional strategies to develop students’ critical 
thinking, problem solving, and performance skills. 


20% 


30% 


50% 


The teacher understands individual and group motivation 






10 


and behavior and creates a learning environment that 
encourages positive social interaction, active engagement 
in learning, and self-motivation. 






100% 


The teacher uses a variety of communication techniques 


1 


4 


5 


including verbal, nonverbal, and media to foster inquiry, 
collaboration, and supportive interaction in and beyond 
the classroom. 


10% 


40% 


50% 


The teacher plans and prepares instruction based upon 


2 


7 


1 


knowledge of subject matter, students, the community, 
and curriculum goals. 


20% 


70% 


10% 


The teacher understands, uses, and interprets formal and 


5 


3 


2 


informal assessment strategies to evaluate and advance 
student performance and to determine program 
effectiveness. 


50% 


30% 


20% 


The teacher is a reflective practitioner who demonstrates a 




1 


9 


commitment to professional standards and is continuously 
engaged in purposeful mastery of the art and science of 
teaching. 




10% 


90% 


The teacher interacts in a professional, effective manner 


4 


3 


3 


with colleagues, parents, and other members of the 
community to support students’ learning and well-being. 


40% 


30% 


30% 
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Appendix A 

Case Analysis Standards 



Issue Identification 

Standard: The teacher considers different theories of motivation and management when confronted 
with classroom motivation and management problems in order to identify all significant issues and 
aspects relevant to both simple and complex problem situations. 

Analysis 

Standard: The teacher analyzes teacher-student interactions in complex motivation and management 
situations using theories of motivation and management to identify needed adjustments to enhance 
social relationships, student motivation and engagement in learning, or to maintain a safe and 
effective learning environment. 

Alternative Strategies 

Standard: The teacher selects and uses effective strategies drawn from the professional literature 
for handling motivation and management problems and to make adjustments to practice. 

Rationale 

Standard: The teacher uses the professional literature on motivation and management to provide a 
theoretical rationale and research support for educational practices. 

Format & Usage 

Standard: The teacher adheres to format requirements and models professional (standard) English 
usage in writing. 
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Appendix B 

Case Analysis Guidelines -Elementary 



Portfolio Entry 

As part of the requirements for EDUC 302 Motivation and Management, you will complete a Case 
Analysis to be included as a portfolio entry in your Teacher Education Portfolio. This handout 
describes the requirements for your case analysis. 

Case Analysis Assignment 

For your case analysis, read the assigned case involving student motivation and classroom 
management issues. Conduct an independent analysis of the case using the attached matrices [see 
attached matrices 1 to 4]. Using Matrix #1 identify: (1) the significant issues related to motivation 
and the creation of effective learning environments warranted by the case description (Issue 
Identification), (2) the actions taken (or inaction) by the teacher or students, and (3) your analysis of 
the issues based on motivation and management course principles (cite the principles). Then, 
using Matrix #2, for the same issues you identified in Matrix #1, (4) provide alternative strategies, 
and (5) provide a rationale for each alternative strategy you suggest and justify your belief in its 
probable effects using the motivation and management principles discussed in EDUC 302 (cite 
them). Do the same thing for issues related to student conduct and misbehavior using Matrix #3 
and Matrix #4. See attached copy of the scoring rubric for this assignment. 

Assigned Case 

Case 1: Marsha Warren in Silverman, R., Welty, W., & Lyon, S. (1996). Case studies for 
teacher problem solving (second edition). New York: McGraw-Hill. This book is on reserve in the 
Oboler Library. 

Targeted Core Standards 

The specific standards and performance indicators for the case analysis assignment are contained in 
your copy of the scoring rubric (see separate handout). This assignment will give you the 
opportunity to demonstrate your ability to meet the following targeted core standards of the Teacher 
Education Program: 

Standard 2: Professional Studies and Research. The teacher uses alternative theoretical perspectives 
and research to guide instructional decision-making and reflection on practice. 

Standard 7: Management of the Learning Environment. The teacher creates and maintains a safe 
and effective learning environment. 

Content Directions 



Provide an insightful overview of the general case situation and the teacher and student 
characteristics (do not exceed one double-spaced page). Then, using your own matrices modeled 
after the ones attached (single spaced word-processed), identify the significant issues and problems 
in your assigned case. Matrix #1 addresses motivation and prevention. Matrix #3 addresses 
classroom management and student misbehavior. Use Matrix #2 and Matrix #4 to provide your 
alternative strategies and your rationale for suggesting them. Close your paper with a one-page 
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section stating any additional conclusions you reached regarding the case overall and the insights 
you gained from this assignment that are applicable to your own teaching (do not exceed one 
double-spaced word-processed page). 

Format Directions 



Your case analysis should be complete, but it cannot exceed 1 1 pages total [1 cover page with 
overview, a maximum of 8 matrix pages, 1 conclusions page and 1 references page]. You must use 
standard 12 point typeface for the first and last page of your report. You may use 10 point typeface 
inside each matrix. The following fonts are preferred: Palatino, Arial, Geneva, or Times New 
Roman. The margins must be one inch on all sides, except for the matrix pages (follow the models 
supplied). Do NOT put your paper in any kind of cover or binder. Staple all pages together in 
the upper left hand corner (do not use a paper clip). Type your name, section number, and 
the date in the upper right hand comer of the first page (the overview page). Title your assignment 
with the name of the case you analyzed (centered on the first page— which is the same page as your 
overview of the case). 

To receive full credit, your case analysis must be well organized and well written. Your grammar, 
usage, expression of ideas, and editorial style should be consistent with the APA Style Manual (4th 
edition). Citation of references should also follow APA style. Be sure to proof read your paper to 
eliminate spelling and punctuation errors. 

Due Date 



Your case analysis is due: 
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